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Summary 


Since 2012 the 10 regional educational laboratories (RELs) have built and supported 
research alliances on topics important to states and local school districts. Research alli- 
ances are sustained collaborations among researchers, administrators, policymakers, and 
practitioners. The broad goal of research alliances is to increase state and local capacity 
to use data and research to inform decisionmaking in a designated education priority area. 

A key task for each research alliance is to establish a set of research questions — a research 
agenda — to guide its work over several years. Setting a research agenda is not an easy task 
in almost any circumstance. It requires assessing what is known about a problem, identi- 
fying the important unanswered questions, and weighing opportunities and constraints, 
including the availability of data, willing research partners, and funding. For research 
alliances, setting a research agenda also can require technical assistance in developing 
researchable questions and negotiating competing interests among the states and districts 
represented at the table. 

This report describes the approach that REL Northeast and Islands (REL-NEI) used to 
guide its eight research alliances toward collaboratively identifying and agreeing on a 
shared research agenda. A key feature of REL-NEI’s approach was a series of two workshops 
for each alliance. The workshops helped groups of individuals with varying backgrounds 
establish an agreed set of questions to guide research with implications for education policy 
and practice. 

This report explains how REL-NEI conceptualized and organized the workshops, planned 
the logistics, accommodated geographic distance among alliance members, developed and 
used materials (including modifications for different audiences and for a virtual platform), 
and created a formal research agenda after the workshops. Materials used for the workshops, 
including facilitator and participant guides to the workshop and workshop slide decks, will 
be available for download from the Tools & Resources section of the REL-NEI website 
(www.relnei.org/tools-resources.html) or by contacting REL-NEI at relneiinfo@edc.org. 



The intended audience for this report is other groups of researchers, practitioners, and 
policymakers who want to work in a sustained partnership on research questions that arise 
from education practice. Although there is no single way to develop and nurture a strong, 
productive research alliance, the lessons from the experience of the RELs may spark new 
ideas and help others avoid pitfalls. The workshop materials developed along with this 
report are customizable to the needs of other alliances. 
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Since 2012 the 10 regional educational laboratories (RELs) have built and supported 
research alliances on topics important to states and local school districts. Research alli- 
ances are sustained collaborations among researchers, administrators, policymakers, and 
practitioners. The broad goal of research alliances is to increase state and local capacity 
to use data and research to inform decisionmaking in a designated education priority area. 


A key task for each research alliance is to establish a set of research questions — a research 
agenda — to guide its work over several years. Setting a research agenda is not an easy task 
in almost any circumstance. It requires assessing what is known about a problem, identifying 
the important unanswered questions, and weighing opportunities and constraints, including 
the availability of data, willing research partners, and funding. For research alliances, setting 
a research agenda also can require technical assistance in developing researchable questions 
and negotiating competing interests among the states and districts represented at the table. 

This report describes the approach that REL Northeast and Islands (REL-NEI) used to 
guide its eight research alliances toward collaboratively identifying and agreeing on a 
shared research agenda. A key feature of REL-NEI’s approach was a series of four workshops 
for each alliance. These workshops helped groups of individuals of varying backgrounds 
establish an agreed set of questions to guide research with implications for education policy 
and practice. 


A key task for each 
research alliance 
is to establish a 
set of research 
questions-a 
research agenda— 
to guide their work 
over several years 


This report provides an introduction and links to the workshop materials, which are avail- 
able for adaptation and use. The report has eight sections: 

• Research alliances: How are they organized, and who participates? 

• What is the purpose of Research Agenda Workshops? 

• Using the workshop agenda and materials to engage, instruct, and prioritize. 

• Planning and leading a virtual Research Agenda Workshop. 

• Example: One alliance’s experience with the workshop. 

• After the workshop: Toward a final alliance research agenda and work plan. 

• Successes. 

• Remaining challenges. 


Research alliances: How are they organized, and who participates? 


REL-NEI established eight research alliances focused on four education-related topics 
identified as regional priorities through a needs-assessment process. Table 1 lists the topics, 
the research alliances aligned with each topic, whether state and district staff participate, 
and the jurisdictions they serve. 

The REL-NEI research alliances are structured with three levels of membership, each 
requiring a different level of commitment: the core planning group, participating members, 
and interested stakeholders. 

• Core planning group; During the first year of working with the alliances, REL-NEI 
focused on recruiting and developing a core planning group (CPG) for each alli- 
ance. The CPG governs the alliance, setting priorities and guiding the agenda and 
project development. Each alliance has 10-15 CPG members who meet regular- 
ly, advise on projects, and generate ideas for alliance events. CPG members were 
asked to commit at least a year to this role. 
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Table 1. The eight Regional Educational Laboratory Northeast & Islands research alliances in 2012 — a 
closer look 



Member composition 
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Topic 1: Identifying and retaining effective teachers and principals 

Northeast Educator Effectiveness 

Research Alliance ✓ ✓ ✓ 

✓ ✓ 

✓ 

✓ 

✓ 


✓ 


✓ 

Topic 2: Improving low-performing schools and districts 

Northeast Rural Districts 

Research Alliance ✓ ✓ 

✓ 



✓ 




✓ 

Urban Schools Improvement 

Alliance ✓ 

✓ ✓ 

✓ 


✓ 


✓ 



Topic 3: Ensuring educational equity and supporting special populations 

Early Childhood Education 

Research Alliance ✓ ✓ 

✓ ✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

English Language Learners Alliance ✓ ✓ 

✓ ✓ 

✓ 


✓ 


✓ 



Topic 4: Increasing college and career readiness, access, 

and completion 








College and Career Readiness 

Research Alliance ✓ ✓ ✓ 

✓ ✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

✓ 

Puerto Rico Research Alliance for 

Dropout Prevention ✓ ✓ 





✓ 




Virgin Islands College and Career 
Readiness Research Alliance ✓ 







✓ 


a. Includes institutions of higher education, education service agencies, other state departments, and the like. 




Source: Authors. 










Figure 1. Regional Educational Laboratory Northeast & Islands alliance structure 


Interested 

stakeholders 
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• Participating members: After REL-NEI had identified members of the CPG, it 
began to recruit participating members in the alliances. Participating members are 
peers or near-peers of the CPG members. For example, the Urban School Improve- 
ment Alliance CPG comprises research directors for medium-size urban districts, 
and participating members were recruited from the research staff at other medi- 
um-size urban districts in the region; other staff in these districts whose respon- 
sibilities may include research, assessment, or evaluation; and state staff involved 
in research, data analysis, and related activities. Participating members attend alli- 
ance convenings, workshops, and events. They also may participate as partners or 
advisors on projects and studies that come from the alliance research agenda. 


• Interested stakeholders: The final layer of membership for each alliance is 
interested stakeholders, which include peers from outside the region, researchers 
focused on the alliance topic, and a national audience of stakeholders. Interested 
stakeholders are included in all dissemination efforts and events. 

Each REL-NEI alliance also has a facilitator and at least one researcher, who together 
make up the alliance team. The alliance team members are specialists in the alliance 
content area. The facilitators, with extensive experience providing technical assistance to 
states and districts and facilitating conversations with education leaders at multiple levels, 
are skilled at synthesizing group discussions, moving discussions forward, and understand- 
ing education practitioner needs. The alliance researchers hold doctoral degrees, bringing 
a broad understanding of research methods and design to the alliance. They draw on this 
expertise during the Research Agenda Workshops — to lead the discussion of research types 
and designs, take extensive notes, and help alliance members synthesize their research 
topics and questions during the workshop brainstorming sessions. 


Since the core 
planning group 
members set the 
direction of each 
alliance, it made 
sense to limit 
participation in the 
Research Agenda 
Workshops to this 
leadership group 


The alliance facilitators and researchers are REL-NEI staff members who have dedicated 
time to work on the design and implementation of activities that nurture the relation- 
ships within each alliance. These activities include convenings, meeting materials that 
inform discussion, shared online space for alliance members, recruitment of new members, 
one-on-one discussions with CPG members, and large virtual and face-to-face events with 
expert speakers and researchers that address alliance interests. 

What is the purpose of Research Agenda Workshops? 


REL-NEI designed the Research Agenda Workshops to engage the 10-15 members of the 
CPG of each research alliance in a collaborative process for identifying shared research 
questions that will form a research agenda for three to five years. Since the CPG members 
set the direction of each alliance, it made sense to limit participation in the workshops to 
this leadership group. 

Clarifying the relationships among research questions, research design, data availability, and study 
feasibility 

A foundational assumption of research alliances is that education practitioners and policy- 
makers identify “problems of practice” in the course of their work and that these problems 
can form the basis of a research agenda. However, many practitioners, immersed in the 
complex challenges of their daily work, struggle to formulate researchable questions to 
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inform the problems they face. Sometimes, they do not understand how research questions 
are related to the methodology. Other times, they struggle to break down overarching 
interests or goals into a set of related projects and studies that might make up a research 
agenda. This is where the skills of researchers become critical to the process. 

To help CPG alliance members understand the relationships among research questions, 
research design, data availability, and study feasibility, REL-NEI designed the Research 
Agenda Workshop to focus on identifying strong research questions and establishing a 
research agenda. These workshops are intended to build these alliance members’ research 
capacity and to structure and scaffold the process of engaging alliance members in devel- 
oping a coherent research agenda that guides their work with REL-NEI. 

The workshops were also developed as a low-risk activity for building relationships among 
CPG alliance members and between alliance members and REL-NEI staff. The workshop 
activities and format were created to prompt participants to express the education-related 
priorities and values that guide them and the work of their organizations. As participants 
clarify and advocate for their research interests, they develop shared understandings of 
each person’s role in the alliance and identify shared beliefs about education. By the work- 
shop’s end, the research agenda should reflect these shared values and become part of the 
identity of the alliance as it grows beyond its founding members. 

Tailoring Research Agenda Workshop logistics to alliance member needs 

The Research Agenda Workshop was designed as two half-day sessions conducted either 
in person or online. For scheduling reasons, however, some alliances needed to hold the 
workshop in a single, day-long session. 

In the first session, the workshop usually addressed content related to research, research 
agendas, and prioritization of topics. The alliances that did the workshop in two sessions 
had participants individually develop research questions as “homework” to complete after 
the first session. When the alliance reconvened in the second session, the questions that 
participants developed were refined in groups. Alliances that met for a single-day session 
had participants develop their research questions individually or in small groups. 

Regardless of whether the workshop was held in a one-day or a two-session format, it was 
challenging to devote enough time to fully present the content and engage in the prioriti- 
zation processes while also balancing the busy schedules of CPG members. Some alliances 
found it difficult to devote a full five or six hours to the workshop. However, when the 
workshop was shortened, alliance leaders found that the groups were less likely to make it 
to prioritizing questions. Participants needed time to work through the processes of focus- 
ing on the types of research they might want to do; identifying and prioritizing research 
topics; and developing, refining, and prioritizing research questions. 

Two other format issues were considered by alliances in planning the workshops: whether 
to build the workshop into part of another meeting or to make it the standalone focus of 
the alliance’s meeting and whether to conduct the workshop in person or as a webinar. 
Alliances that built the workshop into another meeting found it important to ensure a clear 
transition between segments of the meeting. One of the benefits of a face-to-face meeting 
is that participants were extracted from their regular day and could focus exclusively on 


To help core 
planning group 
alliance members 
understand the 
relationships 
among research 
questions, 
research design, 
data availability, 
and study 
feasibility, REL- 
NEI designed the 
Research Agenda 
Workshop to focus 
on identifying 
strong research 
questions and 
establishing a 
research agenda 


4 


the workshop and its agenda. However, scheduling a time for alliance members to meet in 
person for a full day was difficult. 

When deciding between the in-person or webinar format, the alliances’ group cohesion 
and familiarity were also considered. Webinars can work well if the participants know 
each other or have met virtually or in person. For example, the Northeast Rural Districts 
Research Alliance includes practitioners and researchers who had little interaction before 
the alliance was formed. Therefore, alliance members felt it worthwhile to travel across the 
region for a full-day, face-to-face workshop. Knowing whether participants were likely to 
speak up and engage in conversation over the phone was another important consideration, 
as was whether the participants were likely to become distracted. Another consideration 
was whether participants would have the computer and Internet connections needed to 
participate in a webinar. 

Using the workshop agenda and materials to engage, instruct, and prioritize 


A core agenda and set of materials was developed to ensure that the CPG of each alliance 
had some common information and content and to maintain a consistent process across 
the alliances for achieving the workshop goals. Some of the materials and the agenda 
were then customized to suit the audience’s experience and background and the logis- 
tics were tailored as well. The core set of materials, including facilitator and participant 
guides and slide deck, will be available for download from the Tools & Resources section 
of the REL-NEI website (www.relnei.org/tools-resources.html) or by contacting REL-NEI 
at relneiinfo@edc.org. The workshop toolkit can be downloaded and adapted for use in 
other settings. The workshop elements that were maintained across alliances are discussed 
below, as is how the materials were tailored to fit participant backgrounds and interests. 

The core agenda 

The core agenda included 12 segments: 

• Welcome and purpose. 

• Reviewing alliance goal and identifying possible research topics. 

• Reviewing types of research and evidence. 

• Investigating research examples. 

• Reviewing the properties of research questions and agendas. 

• Prioritizing research topics. 

• Generating research questions. 

• Sharing research questions generated. 

• Refining questions. 

• Prioritizing research questions. 

• Building the research agenda. 

• Closing and next steps. 

Elements in common 

Some elements of the agenda and materials were standard across every workshop. These 
elements were: 

• Prework to be completed in preparation for the workshop. 

• An overview of REL work and an explanation of the workshop’s purpose. 


When deciding 
between the 
in-person or 
webinar format, 
the alliances’ 
group cohesion and 
familiarity were 
also considered 
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• A emphasis on the alliance goal and on individual and group brainstorming of 
possible research topics. 

• Group prioritization of possible research topics. 

• A review of various types of research and levels of evidence. 

• An overview of the properties of research questions and research agendas. 

• Individual and group exercises for generating research questions and refining them. 

• Group discussion of priority research questions. 

While these elements were common across the workshops, the way each element was addressed 
differed by alliance. A workshop planning team developed the core materials and worked with 
alliance teams to customize them to each alliance context. In customizing the workshops, it 
was important that workshop planners — who then trained the alliance teams to conduct the 
workshops — consider participants’ backgrounds, experiences, and current positions. 

The knowledge and relationships that alliance facilitators and researchers had developed 
with their alliance membership helped in tailoring each workshop. An alliance facilitator 
or researcher, for example, could inform the workshop planners that its CPG had members 
who know about research design and who thus do not need a primer on research methods. 
Although every workshop explored types of research and levels of evidence, the amount 
of time and depth varied by alliance members’ needs. For example, the Urban School 
Improvement Alliance CPG consists of sophisticated users and producers of research 
and data analysis. As a result, their workshop did not include a minidecture on types of 
research and levels of evidence. Instead, the facilitator and researchers led a discussion of 
the CPG members’ experiences with varying types of research. 

The timeframes in which alliance members prioritized research questions developed in 
the workshop were also customized. In each alliance, participants voted on prioritizing 
questions. Participants received a set number of votes that they could distribute across 
the questions to prioritize them. Some alliances were comfortable with having partici- 
pants prioritize questions during the workshop, immediately after question generation and 
refinement. However, other alliances wanted the opportunity to confer with colleagues or 
supervisors at their home agencies or to have some time to reflect before voting. 

The content of the prework assignment, which was intended to get participants thinking 
about research design, research methods, types of research, types of data, research ques- 
tions, and how questions and available data inform design, was also customized for each 
alliance. Links were provided to examples of different types of research studies — from 
descriptive studies to randomized controlled trials — and participants were asked to read 
one of the study summaries before the workshop. 

The research examples that workshop planners provided for each workshop differed by 
alliance topic. While the focus of the prework assignment was to get participants thinking 
about research methods, participants would be more engaged if they read examples from 
topics related to their interests. Thus the College and Career Readiness Research Alliance 
received examples of research on graduation rates, while the English Language Learners 
Alliance received studies on the mobility of English language learner students. 

As a result of these customizations, the facilitator and participant guides and the slide deck 
were slightly different for each alliance. 


The knowledge and 
relationships that 
alliance facilitators 
and researchers 
had developed 
with their alliance 
membership 
helped in tailoring 
each workshop 
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Planning and leading a virtual Research Agenda Workshop 


As noted, the workshops were designed to be conducted in person or as a webinar. The 
workshop materials prepared to accompany this report are tailored to an in-person session 
since many researcher-practitioner alliances will not experience the same geographic 
challenges and travel constraints as the REL-NEI alliances. However, some traditional 
face-to-face workshop facilitation techniques were adapted to a webinar and are discussed 
below for users to consider if they plan to conduct the workshop in a virtual environment. 
Overall, the workshop design strived to maintain the engagement of participants in the 
webinar format, even when they could not move physically into small groups or join in 
other activities that help maintain engagement in an in-person meeting. Three workshop 
processes were modified for webinar format: small-group discussions, whole-group discus- 
sions, and prioritization activities. 

Small-group discussions 

The workshop design used small-group breakout sessions to provide all participants with 
opportunities to share ideas and opinions. Discussion notes from the work done in these 
small groups were then shared in the large group. This work included identifying pos- 
sible research topics and examining research examples. Therefore, the webinar platform 
(Adobe® Connect™) also had to facilitate virtual breakout groups. The groups worked 
well when there were clear directions about the group task, when the groups were not too 
large, and when a facilitator and note-taker were identified. 


The groups 
worked well 
when there were 
clear directions 
about the group 
task, when the 
groups were not 
too large, and 
when a facilitator 
and note-taker 
were identified 


The workshop facilitator had to know in advance who would be in which small groups so 
that the process of electronically moving into the breakout rooms would not confuse the 
participants. Figure 2 shows what webinar breakout sessions might look like. The breakout 
group in the figure is prioritizing topics and research questions within them. The group 
consists of four CPG members and a facilitator. Three other small groups are holding 
similar discussions simultaneously, with the same visual format. While in small groups, 
participants would see only the input from their group on the screen. Once the breakout 
groups are reunited into the full group, the input from each breakout group would be dis- 
played for a full group discussion. 


Whole-group discussions 

Webinars, like in-person workshops, work better when they are interactive and partici- 
pants are engaged. Facilitated discussions were used in the webinar workshops to engage 
participants in thinking about and sharing their ideas on research topics and questions. 
Workshop facilitators used the “whiteboards” in the webinar platform to take notes, as 
shown in Figure 2. All participants could see these notes. In breakout groups the partici- 
pants were given access to the whiteboards so that the note-taker could be a participant. In 
addition to using the whiteboards to record notes and keep everyone focused on the same 
thing, participants were encouraged to use the Adobe Connect chat function to share 
ideas and comments. To acclimate participants to the chat function, the workshop facili- 
tators conducted a demonstration welcoming each participant to the webinar and asking 
them questions. 
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Figure 2. Examples of webinar breakout sessions 


1) Eval sys impact on tch prac/std outcome 


2) Prof dev for teachers/leaders 


systems? Engaging stakeholders? 

Participant 3 what are the key components of evaluation systems? 
to what degree do these have impact on professional practice 
and student outcomes. 

participants what is the impact of eval system on Aligning HR 
systems? 

Participant 3 what is the impact of evaluation systems on principals? 

Participant 2 How to use student and staff feedback in educator evaluation? 
(e.g. staff feedback on Icadcrs/student feedback on both teachers and 
leaders) What instruments are there to use for those purposes? 

Participarrt4 - the role of self assessment in eval systems/ does a teachers 
self assessment aiigiVconnect with evaluators assessment. 

Participant 4 ■ correlation botweon feedback given in eval systems and 


will be offered? How will district/school-wide pd fit into new systems where 
individual teacher evaluations will be driving pd? 

Partidpant t -what is the effectiveness of this linked pd/how to 
measure the effectiveness of this linked pd? 

Participants Did the pd have an effect on teachers goals (practice and student 
outcomes)? And how to do this (measure quality/links)? 

Participant 4 - how effective is PD in moving teachers through the rating 
systems of eval systems? Effectiveness of PD 

Participant i what pd (and teachers implementation of what they learned in 
PD) is related to student outcomes 

Participant 3 - see this as subtopic of Topic 1 Systems 

Participant 3 comparing state rules/policies and outcomes (see Other 
questions of interest below) 


3) Teacher and leader preparation 


4) Other questions of interest 


regional, and local levels, in developing evaluation systems? 
(may relate to planning and design, in Topic 1) 

■artidpants districts will have own evaluation systems, how can 
we encompass these so that preparation programs can 
adequate prepare/inform their students about the different 
evaluation standards. How to prepare teachers for the 
variation in evaluation systems at the local level. 

Participant4are there examples of state systems vs guidelines for 
impacting teacher/leader preparation (and teaheer evaluation) 


Q -- Does a requirement for yearly checks on the proficiency of evaluators 
improve evaluations? 

- Does the number of evaluees per evaluator impact the quality of evaluations? 

- What's the role (and differences in outcomes) of state policy 
driving a system vs. a district designing their own system? * (Scott) 
~ What are barriers to implementation of new evaluation systems? 

'articipant 5 reliability and validity of evaluation instruments and 
systems **” 

articipant 4Do NBPTS teachers tend to get higher ratings? 


Source: Authors. 


Prioritization activities 

Participants in the in-person workshops used the “Focus Four” prioritization process to 
prioritize research topics and research questions. The benefit of this process is that the pri- 
orities of the alliance members become visually clear to participants. The process includes 
four steps. First, participants brainstorm by identifying topics of interest related to the 
overall goal of the alliance. Second, participants clarify their input in the brainstorming 
step by asking questions and making comments about topics and ideas presented. Third, 
participants are given the chance to advocate for their own or others’ ideas. Fourth, partic- 
ipants canvass or demonstrate their preferences by placing “sticky colored dots” or Post-It 
notes on the topics or questions (written on newsprint) that they think should take pri- 
ority. (For more detailed instructions on running a Focus Four process, see the facilitator 
guide available from the Tools & Resources section of the REL-NEI website (www.relnei. 
org/tools-resources.html) or by contacting REL-NEI at relneiinfo@edc.org.) 
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Figure 3. Example of a webinar poll on priority research topics 

Vote 1 

Vote 1 




O Evaluator training & evaluation reliability 

■ 

40% 

(2) 

O Evaluation and school culture and climate 


0% 

(0) 

O State policy variation and impact 


0% 

(0) 

Educator goals and goal setting (and relationship to teaching) 


0% 

(0) 

O Implementation (fidelity, barriers) 


0% 

(0) 

Multiple rating categories 


0% 

(0) 

O Evaluation systems impact on teacher practice/student outcomes 

mm 

20% 

(1) 

Professional Development 

mm 

40% 

(2) 

O Student learning objectives 


0% 

(0) 

O Evaluating other educators (not classroom teachers) 


0% 

(0) 

O Teacher preparation 


0% 

(0) 

O Leadership/Leader preparation 


0% 

(0) 

Source: Authors. 


The Adobe Connect poll function was used to adapt this method to a webinar. The webinar 
producer took the lists of proposed priority topics and research questions and created online 
polls. Participants could read and select from all the proposed priority topics or research ques- 
tions. The polls were set up so that participants could see the results as people voted. This mim- 
icked the idea of participants seeing the clustering of priorities at an in-person workshop. In the 
Focus Four process participants are usually given several votes, such as half the number of items 
to be voted on plus one. In the webinar format the webinar producer displayed the same poll 
(for example, with all the proposed research topics) up to three times. This gave participants 
the option to vote for a separate priority topic each time or to vote for the same topic up to 
three times. Figure 3 shows what a webinar poll on priority research topics might look like. 

One alliance’s experience with the workshop 


All eight REL-NEI research alliances participated in a Research Agenda Workshop in the 
first year of the REL contract. This section provides a brief description of how one alliance, 
the Urban School Improvement Alliance, experienced this workshop. 

The alliance’s membership includes research directors at medium-size urban school dis- 
tricts in the Northeast, including New Haven (Connecticut), Portland (Maine), Provi- 
dence (Rhode Island), Syracuse (New York), Worcester (Massachusetts), and Yonkers (New 
York). Together, the members established the alliance’s goal as follows: 

The goal of [the Urban School Improvement Alliance] is to help build the capac- 
ity of its local education agency members to use and access data to address ques- 
tions around how to improve low-performing schools. Members are interested in 
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examining school performance in conjunction with larger social, organizational, 
and instructional contexts. 

The alliance chose to hold its workshop in two 2 -hour webinar sessions, on June 14 and 
July 24, 2012. Along with the alliance facilitator and two alliance researchers, six members 
attended the first session and five members attended the second. In the first session partici- 
pants brainstormed 16 initial research agenda topics. Sample topics included: 

• Definition of low-performing schools. 

• Data use at the school and district levels and student outcomes. 

• School-level data use barriers; how have schools overcome them? 

• iPads/technology in schools. 

• Evaluation system and student performance. 

After identifying the 16 initial research topics, alliance members participated in the affin- 
ity grouping process, where they discussed how to organize “like topics” into broad cat- 
egories using Adobe Connect’s whiteboard and chat features. At the conclusion of the 
exercise the group had identified six broad research topics, each with bulleted subtopics: 
low-performing schools, data use, technology, equity, limitations, and evaluation. One alli- 
ance researcher then led the members through a voting process to prioritize the six topics. 
Participants were told they could each cast three votes and could distribute the votes 
however they wanted (meaning they could vote for one topic up to three times). The vote 
was conducted anonymously, using the polling feature. The voting results were as follows: 

• Low-performing schools (6 votes). 

• Technology (4). 

• Data use (3). 

• Evaluation (3). 

• Equity (2). 

• Limitations (0). 

The first workshop session ended after the vote. The alliance facilitator asked the members 
to think about research questions they had in relation to the top vote-getting categories 
and the data that might be available to answer them. Participants were asked to bring 
those questions to the second session (July 24). 

Between the two sessions the REL alliance team sent workshop notes to all the alliance’s 
CPG members, whether they attended the first session or not. The alliance team also sent 
a link to the webinar archive of the session, so that absent members could view it. Based 
on the feedback of members who did not attend the workshop session, data use was identi- 
fied as a higher priority topic than evaluation. 

The alliance facilitator began the second webinar session by reviewing what was covered 
in the first session and presenting the goals of the second. The group then reviewed and 
discussed the three priority research topics: low-performing schools, technology, and data 
use. This discussion proved to be a rich reflection of the research topics, and participants 
appreciated the further conversation on them. 

The session then moved into generating research questions aligned with the priority 
research topics. Participants were asked to share research questions that they want to see 


Reviewing what 
was covered in 
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answered. As participants proposed questions, members of the alliance team organized the 
questions and wrote down clarifying notes on the Adobe Connect whiteboard. 


No one offered a question related to technology. Listed here are some of the proposed 
questions for the other two priority topics: 

• Low-performing schools: 

° What are schools doing to use data to reduce achievement gaps? 

° What are definitions of college' and career-ready? Is there any empirical evi- 
dence to support them? 

o What constitutes needs for remediation in college? Are there commonalities 
across institutions? 

• Data use: 

° What are the impediments to data use on the part of teachers and school 
administrators? 

o What is the link between data use and student and school performance? 

° What facilitates the use of data to help increase student learning? What are 
the essential capabilities, regardless of budget cuts and the like? 

o How can districts best structure themselves so that data is used well for district 
and policy decisions, including at the school-board level? 

The group discussed the proposed research questions, clarifying and refining them. As the 
session wound down, the alliance facilitator explained the next steps. He told the group 
that the alliance team would further refine the research agenda (two priority research 
topics and associated questions) and then send a draft to alliance members for feedback. 
The group also discussed potential researchers to invite to speak with the group to address 
some of their questions. 


After each 
alliance’s Research 
Agenda Workshop 
was complete, 
debriefing sessions 
were held to 
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After the workshop: Toward a final 
alliance research agenda and work plan 


After each alliance’s Research Agenda Workshop was complete, debriefing sessions were 
held with the workshop facilitators, the REL-NEI director of research, and the REL-NEI 
manager of research and evaluation. The goal of the sessions was to examine the priori- 
tized list of research questions and to develop a proposed work plan of projects that might 
help answer the questions. Each meeting included discussions of what relevant research 
already existed and what events could be hosted to disseminate and discuss that research. 

Questions were identified that required additional data or an investigation of possible 
existing data sources prior to the development of a research study proposal. For example, 
one area of focus that emerged from the Early Childhood Education Research Alliance’s 
workshop was early childhood practices and primary grade outcomes. While state data 
systems were still developing links between prekindergarten and elementary education 
data, it was decided to conduct data catalogs of district-level data in two jurisdictions. This 
work showed some promise in assessing whether a study was feasible. Also explored were 
questions that signaled the need for training to build capacity in research and data use 
(such as questions on teacher use of data to prevent dropouts), which emerged from the 
workshop conducted with the Puerto Rico Research Alliance for Dropout Prevention. 
Finally, each debriefing session included discussions of possible study designs for questions 
that were appropriate for immediate or future research studies. 
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All these discussions focused on the timing and sequencing of projects — to create a fea- 
sible scope of work from year to year and to ensure a consistent flow of work to main- 
tain alliance engagement. Projects and timelines were based on research goals set out for 
overall REL-NEI work. These goals prioritized projects that would report frequently to 
the alliance, engage some alliance members intensively through project-specific advisory 
committees, and result in a quick turnaround of Institute of Education Sciences-approved 
products so that alliance members would not be waiting years for tangible results. Debrief' 
ing sessions reinforced the need to create a coherent body of work for each alliance. For 
alliances with research agendas containing more than one priority topic area, one topic of 
focus was selected for REL-NEI work so that the projects would form a coherent body of 
work. REL'NEI’s work in building capacity supported efforts in all the alliances to engage 
with other external researchers to address parts of the research agenda that will not be the 
focus of the REL'NEI projects. 

Finally, all the alliance research agendas and proposed work plans went through a series 
of reviews and revisions by internal leadership groups focused on additional REL'NEI con- 
cerns. The research strategy team was composed of senior researchers from partner orga- 
nizations and the REL'NEI deputy director, director of research, and manager of research 
and evaluation. The team met monthly to review alliance agendas and proposed work 
plans with a focus on how the proposed work would contribute to the overall advance- 
ment of research in the regional priority areas identified through needs-sensing with 
stakeholders. REL'NEI leadership — including the director, deputy director, director of 
research, and manager of research and evaluation — reviewed the work plans of all the 
alliances as a whole set, with a focus on resource availability and equity across the allianc- 
es and the region in order to develop a proposed scope of work for the second year of the 
REL contract. This final scope of work was reviewed by the REL'NEI Governing Board to 
verify that regional needs were well understood and that the plans to address them were 
appropriate. 


Each alliance 
team presented 
the proposed 
work plan to its 
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for understanding, 
communicate 
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While the alliance’s research agendas and proposed work plans circulated through the 
review process, each alliance team presented the proposed work plan to its members 
to check for understanding, communicate REL-NEI plans, and obtain feedback on the 
research agenda and upcoming projects. The agendas and work plans went through two or 
three reviews and revisions with the alliances and internal review teams before a final draft 
was produced. The research agendas did not change drastically from what had emerged 
from the Research Agenda Workshops, but the priorities were clarified and the questions 
refined. At the end of this process each alliance finalized two documents: a public research 
agenda posted on the alliance webpage that contains all research questions along with 
past and ongoing projects, and an internal agenda and work plan for the alliance CPG 
members that includes details about future projects. 


Successes 


The process of engaging each research alliance in the research agenda-setting process 
resulted in five types of successes. The internal alliance teams communicated often with 
REL-NEI leadership and one another about what they learned from the workshops and 
how to improve the process. What follows is a summary of these conversations. 
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Authentic dialogue 


Many alliance CPG members had never met or worked with one another, and there was 
concern that the lack of familiarity would make it difficult to foster workshop discussions. 
The workshop was thus designed to include a number of guided discussion prompts that 
would get members of the group talking. These discussions started with general brain- 
storming, allowing participants to familiarize each other with the broad issues. The discus- 
sions then narrowed to finding commonalities and differences and finally to consensus on 
the most pressing issues. 

Alliance teams were trained in promoting these discussions by walking them through the 
workshop exercises and simulating the activities. The REL-NEI researchers who developed 
the workshop design and materials modeled the desired facilitation skills and encouraged 
the alliance facilitators and researchers to take ownership of the workshop content, mate- 
rials, and activities. These trainings typically took about two hours, which allowed enough 
time to get through the entire workshop outline. The trainings were scheduled to give 
each alliance team an opportunity to adapt the materials to their alliance needs before the 
actual workshop. 

Common language 

Because the majority of the REL-NEI alliances have members from different states and 
jurisdictions, it was necessary to ensure that all participants were using the same defini- 
tions and terminology. Alliance facilitators and researchers listened for jargon and pressed 
the participants to define terms they were using. The discussions that came from interac- 
tions helped participants deepen their knowledge of their colleagues’ issues and circum- 
stances and negotiate common definitions for words or phrases used differently across the 
different geographies. 

Research agenda product 

The result of each workshop was a tangible research agenda document that was shared 
with the alliance members and that will guide future work. These documents allow record- 
ing of the research process and tracking of progress, sharing of goals and research questions 
with future members and the region, and annual review of the agenda with the alliance 
members. Because the alliance members co-created these documents, they have a sense 
of ownership over the work conducted in the service of the alliance. The documents also 
help ensure that no alliance work falls outside the goals and questions documented by the 
research agenda. 

Input to needs-sensing 

While the ultimate goal of the Research Agenda Workshops was to develop a coherent 
research agenda, alliance members discussed a number of important issues. By allowing them 
time to discuss these issues — both those that directly relate to the research agenda and those 
that do not — the workshops helped everyone better understand education practice in the 
region. Because of this, the internal REL-NEI team continues to build knowledge on how to 
make its work with alliance members useful, relevant, and timely. All the issues and needs 
raised by alliance members were receorded in the REL-NEI needs-sensing database. 


The result of each 
workshop was a 
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Relationship-building 


The process of developing the research agendas was the first step in getting alliance 
members to build collegial relationships around a common mission. One of the consistent 
pieces of feedback received in the evaluations of these sessions was how much participants 
appreciated the opportunity to engage with their peers and colleagues on topics of urgency. 
Many examples were noted of alliance members raising an issue during these workshops 
and receiving tools, documents, or advice from others in the group to help them proceed 
with their work. 


Remaining challenges 


As work with the REL-NEI research alliances moves forward, several challenges remain. 
Four challenges are briefly described below. 

Addressing changing membership 

REL'NEI alliances have already experienced some turnover as people retire, move out of 
the region, or change positions. REL'NEI is planning on expanding alliance memberships 
in future years to include different stakeholder groups and interests. Because of this fluid- 
ity, one issue to consider is how to orient new members to the research agendas and give 
them a sense of ownership, while maintaining the goal of creating and implementing a 
coherent agenda within each alliance. 

Balancing being nimble with cohesion 


While the theme 
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often in education 
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research in practice 


Education landscapes will likely change for participants as a result of elections, federal 
policy initiatives, or other state or district factors. How to maintain the goal of creating 
coherent research agendas while still being responsive to changing contexts and needs in 
the region will have to be explored. For example, the granting of federal Race to the Top 
early childhood contracts to some states has made early education a higher priority than it 
was at the formation of the Early Childhood Education Research Alliance. 


Measuring impact 

Systems need to be developed to track progress against the research agendas and to deter- 
mine the impact this work is having for members of each alliance and stakeholders across 
the region. Efforts are under way to define what a successful alliance is and how to know 
that an alliance’s research agenda is driven by member needs and directly relevant to 
their work. The research and data-use capacity that the alliance work builds in alliance 
members needs to be measured. Although secondary to the focus on serving stakeholders, 
it is important to contribute to the understanding of alliance research and the develop- 
ment of a typology of alliances and collaborative practices. While the theme of collab- 
orative research is cited often in education research, it is important that the field move 
beyond the use of labels to a better understanding of how to identify and catalog collabora- 
tive research in practice so that the labels do not lose meaning. 


14 


Matching program timeline to alliance needs 


Timelines associated with the REL-NEI program required that the alliance create research 
agendas and develop projects that relate to them within six months of its first meeting. 
Due to this compressed timeline, the development of the alliance research agendas was 
highly structured and relatively quick. Some alliance members would have preferred a 
more extended discussion and iterative process. Many of the models of alliance research 
have started more organically, with time to develop relationships and create agendas over 
several years. Given the five-year framework of the REL contract, the organic process of 
relationship-building among some existing research alliances has to be balanced against 
the structured requirements of the contract. 


* * * 

As stated in the introduction, the major thrust of the new RELs is to launch and support 
a set of research alliances consisting of state and local policymakers, practitioners, and 
researchers — to undertake research on areas of special concern in each large and diverse 
REL region. The Research Agenda Workshop was created to support alliances in the iden- 
tification and development of research agendas rooted in their actual needs and priorities. 

The workshop process proved useful for generating a realistic and attainable research 
agenda with buy-in from alliance members. It showed that attaining consensus around 
research priorities and questions was possible and that sharing the workshop experience 
brought alliance members closer. 


The workshop 
process showed 
that attaining 
consensus around 
research priorities 
and questions 
was possible 
and that sharing 
the workshop 
experience 
brought alliance 
members closer 


It is hoped that the experience outlined here and the materials developed can be of use to 
other RELs and others seeking to create research agendas among alliances or other types 
of research-practitioner partnerships. Every research alliance is unique, however, so users 
are encouraged to customize these materials and the workshop to meet the distinct needs 
of their group. 
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